1. Field of the Invention
The present invention relates to techniques for calculating, using captured images of markers existing in a real space, placement information regarding the markers or a measurement target or a parameter of an image pickup unit.
2. Description of the Related Art
Extensive research has been conducted on the mixed reality (MR) technology that superposes virtual space information (e.g., virtual objects rendered by computer graphics (CG) or text information) on a real space in real time and presents the superposed result to a user. An image display apparatus used in such an MR system is mainly implemented by a video see-through method of superposing an image in a virtual space, which is generated in accordance with the position and orientation of an image pickup device, such as a video camera, on an image in a real space, which is captured by the image pickup device, and rendering and displaying the superposed images.
To enable the user to use the MR system without any uncomfortable feeling, an important factor lies in how accurately the real space is registered with the virtual space. Many measures have been taken to achieve the accurate registration. In general, the registration problem in MR applications eventually leads to the problem of obtaining the position and orientation of the image pickup device relative to a space or an object onto which virtual information is to be superposed.
To solve the problem, registration techniques using markers placed or set in an environment or on an object are disclosed in “Fukugo-genjitsukan ni okeru ichiawase shuhou (A review of registration techniques in mixed reality)” by Sato and Tamura, Collected Papers I of Meeting on Image Recognition and Understanding (MIRU 2002), Information Processing Society of Japan (IPSJ) Symposium Series, vol. 2002, no. 11, pp. I.61-I.68, 2002 (hereinafter referred to as “document 1”). In these techniques, the three-dimensional coordinates of each maker are given in advance. The position and orientation of an image pickup device are computed using the relationship between the preliminary-given three-dimensional coordinates of each marker and the image coordinates of each marker within an image captured by the image pickup device.
In “Fukugo-genjitsukan no tame no haiburiddo ichiawase shuhou-6 jiyudo sensa to bijon shuhou no heiyou—(A robust registration method for merging real and virtual worlds-combining 6 degree of freedom (DOF) sensor and vision algorithm)” by Uchiyama, Yamamoto, and Tamura, Collected Papers of the Virtual Reality Society of Japan, vol. 8, no. 1, pp. 119-125, 2003 (hereinafter referred to as “document 2”), a hybrid approach utilizing markers and a magnetic or optical 6-DOF position/orientation sensor is disclosed. Although the 6-DOF position/orientation sensor provides measured values in a stable manner, the accuracy is not sufficient in many cases. Therefore, in this method, the position and orientation of an image pickup device, which are obtained from the 6-DOF position/orientation sensor mounted on the image pickup device, are corrected using image information regarding the markers, thereby improving the accuracy.
In “UG+B hou: shukan oyobi kyakkan shiten kamera to shisei-sensa wo mochiita ichiawase shuhou (UG+B: A registration framework using subjective and objective view camera and orientation sensor) by Sato, Uchiyama, and Yamamoto, Collected Papers of the Virtual Reality Society of Japan, vol. 10, no. 3, pp. 391-400, 2005 (hereinafter referred to as “document 3”), a 3-DOF orientation sensor is mounted on an image pickup device, and a method of measuring the position and orientation of the image pickup device using image information regarding markers and an orientation value measured by the orientation sensor is disclosed.
In the above-described registration methods using the markers, three-dimensional information regarding the markers (hereinafter referred to as “placement information regarding the markers” or simply as “marker placement information”) in a three-dimensional coordinate system serving as a registration reference (hereinafter referred to as a “reference coordinate system”) needs to be obtained in advance. The placement information regarding the markers can be manually measured using a ruler, a protractor, a meter, and the like. However, such manual measurement involves many steps but has poor accuracy. Thus, the marker placement information has been obtained in a simple, highly accurate manner using image information.
Placement information regarding markers, each of the markers being represented as the position of a point in a three-dimensional space (hereinafter referred to as “point markers”), can be obtained using a bundle adjustment method. The bundle adjustment method is a method of simultaneously calculating, on the basis of many images captured from various directions, the positions of a group of points in the space and the position and orientation of the image pickup device at the time the image pickup device has captured each of the images. More specifically, the method optimizes the positions of the group of points and the position and orientation of the image pickup device so as to minimize the sum of errors between the observed positions of the points in each of the captured images and the calculated positions of the points in the image, which are calculated on the basis of the positions of the points and the position and orientation of the image pickup device. In contrast, as in the case of a marker with a two-dimensional shape, such as a square marker, whose placement information is represented by the position and orientation in the reference coordinate system (hereinafter referred to as a “two-dimensional marker”), the bundle adjustment method, which is the method of obtaining the positions of the group of points, cannot be applied directly. Therefore, a method of obtaining placement information regarding two-dimensional markers and point markers, which is similar to the known bundle adjustment method, is disclosed in “Maka haichi ni kansuru senkenteki chishiki wo riyoushita maka kyaribureishon houhou (A marker calibration method utilizing a priori knowledge on marker arrangement)” by Kotake, Uchiyama, and Yamamoto, Collected Papers of the Virtual Reality Society of Japan, vol. 10, no. 3, pp. 401-410, 2005 (hereinafter referred to as document 4).
In the above-described registration methods using the markers and the 6-DOF position/orientation sensor or the 3-DOF orientation sensor, not only the placement information regarding the markers, but also placement information regarding the sensor must be measured in advance.
For example, in the case that the magnetic 6-DOF position/orientation sensor is used, a transmitter is fixed in the space, and a receiver is mounted on a measurement target (e.g., an image pickup device) to measure the position and orientation of the measurement target. The sensor is configured to measure the 6-DOF position/orientation of the receiver in a coordinate system defined by the transmitter. Thus, to obtain the position and orientation of the measurement target in the reference coordinate system, placement information (that is, the position and orientation) of the transmitter relative to the reference coordinate system and placement information (that is, the position and orientation) of the receiver relative to the measurement target must be measured in advance. Japanese Patent Laid-Open No. 2003-269913 (corresponding to U.S. Pat. No. 6,792,370) discloses a method of obtaining placement information of the sensor using a plurality of images of markers placed in the reference coordinate system, which are captured from various directions. In the case that the 3-DOF orientation sensor is used, the orientation sensor is mounted on a measurement target (e.g., an image pickup device) to measure the orientation of the measurement target. To this end, placement information (that is, the orientation) of the orientation sensor relative to the measurement target needs to be measured in advance. Japanese Patent Laid-Open No. 2005-326275 (corresponding to U.S. Published Application No. 2005/0253871) discloses a method of obtaining the orientation of the sensor relative to the measurement target using a plurality of captured images of markers.
In the related art, many images captured from various directions have been used to measure the placement information regarding the markers for registration and the sensor. Normally, these images are manually captured by the user who decides the image capturing positions. However, such random capturing of the images is insufficient for highly accurate measurement of the placement information. The user must be fully experienced and have enough knowledge. In other words, not everyone can easily measure the placement information.
Since the known measurement of the placement information regarding the markers and the sensor requires time-consuming preparations, the measurement must be conducted before the user experiences the MR system. Therefore, no new markers can be added to expand the moving range while the user is experiencing the MR system. When the marker or sensor placement information changes while the user is experiencing the MR system, no actions can be taken in real time to handle such a change.